Reproducing published deep learning papers to validate their conclusions can be difficult due to sources of irreproducibility. We investigate the impact that implementation factors have on the results and how they affect reproducibility of deep learning studies. Three deep learning experiments were ran five times each on 13 different hardware environments and four different software environments. The analysis of the 780 combined results showed that there was a greater than 6% accuracy range on the same deterministic examples introduced from hardware or software environment variations alone. To account for these implementation factors, researchers should run their experiments multiple times in different hardware and software environments to verify their conclusions are not affected.

Cybersecurity researchers on Tuesday disclosed a new large-scale campaign targeting Kubeflow deployments to run malicious cryptocurrency mining containers. The campaign involved deploying TensorFlow pods on Kubernetes clusters, with the pods running legitimate TensorFlow images from the official Docker Hub account. However, the container images were configured to execute rogue commands that mine cryptocurrency. Microsoft said the deployments witnessed an uptick towards the end of May. Kubeflow is an open-source machine learning platform designed to deploy machine learning workflows on Kubernetes, an orchestration service used for managing and scaling containerized workloads across a cluster of machines.

For this post, we measured fine tuning performance (training and inference) for the BERT (Bidirectional Encoder Representations from Transformers) implementation in TensorFlow using NVIDIA Quadro RTX 8000 GPUs. For testing, we used an Exxact Valence Workstation fitted with 4x Quadro RTX 8000's with NVLink, giving us 192 GB of GPU memory for our system. These tests measure performance for a popular use case for BERT and NLP in general, and are meant to show typical GPU performance for such a task. We made slight modifications to the training benchmark script to get the larger batch size metrics. The script runs multiple tests on the SQuAD v1.1 dataset using batch sizes 1, 2, 4, 8, 16, 32, and 64 for training, and 1, 2, 4, and 8 for inference.